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FIELD OF THE INVENTION 



The present invention relates generally to field of user interfaces for 
computer systems. More specifically, the present invention is directed to a 
5 method and an apparatus for combining the user interfaces of a plurality of 
applications. 

BACKGROUND 

10 User interfaces refers to the methods and devices that are used to 

accommodate interaction between the machines and the users who use them. 
The user interfaces allow for communicating information from the machine to 
the user, and communicating information from the user to the machine. 

Generally, the user interacts with the machine through the user interfaces 

15 presented by an application running on the machine. The user interfaces are 
designed by the software developers with the purpose of allowing the user to 
take full advantage of the functions of the applications. This may include 
presenting a finite number of options for the user to choose rather than 
requiring the user to memorize and manually enter commands from a large 

20 number of command options. Furthermore, the user interfaces are generally 
designed so that they are intuitive and easy to use. This significantly reduces 
the training to use a new application allowing the user to become productive in 
a short time. 

Since the user may use applications designed by different groups of 
25 software developers from the different software vendors, the user is required to 
be familiar with multiple user interfaces and to interact with each interface 
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individually. This requirement still exists even though there may be 
overlapping interfaces among the applications. 

Unless there are collaborations among the software vendors, the user 
interfaces for each application are designed virtually without any anticipation of 
being modified by applications from other software vendors. This is because the 
software vendors want to preserve the carefully designed graphics and layout of 
their user interfaces. As a result, the user is unable to take advantage of the 
common user interfaces and data structure among the applications. 
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SUMMARY OF THE INVENTION 



In one embodiment, a method for combining the user interfaces of several 
applications is disclosed. Data generated by a first application is extracted from 
5 a display buffer. The data is associated with a user interface from the first 

application. From the extracted data, a layout pattern is recognized. Using the 
layout, an overlay is created. The overlay is used to display a second data 
generated by a second application. There is no direct link between the first 
application and the second application. 



15 



20 



25 
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BRIEF DESCRIPTION OF THE DRAWINGS 



The present invention is illustrated by way of example in the following 
drawings in which like references indicate similar elements. The following 
5 drawings disclose various embodiments of the present invention for purposes of 
illustration only and are not intended to limit the scope of the invention. 

Figure 1A illustrates one embodiment of a computer system. 

10 Figure IB is an exemplary flow diagram of one embodiment of the 

melded user interfaces. 

Figure 2 is an exemplary web-based calendar manager illustrating a 
parent application. 

15 

Figure 3A is an exemplary layout of the calendar produced by an edge 
detection operation of a pattern recognition application. 

Figure 3B is an exemplary layout produced by an edge smoothing 
20 operation. 

Figure 4A is an exemplary layout with the boundaries identified. 

Figure 4B is the same layout as in Figure 4A with the corners and 
25 intersections identified. 



Figure 4C is the same lay out as in Figure 4B with the addition of the 
Ricoh 5 074451. PI 15 



information previously compiled about the calendar. 



Figure 5A illustrates the exemplary calendar parent application with the 
overlay information from the child application. 

5 

Figure 5B illustrates an exemplary interaction with the child 
application. 



Figure 6 illustrates one embodiment of a computer-readable medium 
10 containing various sets of instructions, code sequences, configuration 

information, and other data used by a computer or other processing device. 
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DETAILED DESCRIPTION 



A method and apparatus for combining the user interfaces of several 
applications are disclosed. In the following description, for purposes of 
5 explanation, specific nomenclature is set forth to provide a thorough 

understanding of the present invention. However, it will be apparent to one 
skilled in the art that these specific details are not required in order to practice 
the present invention. 

Some portions of the detailed descriptions that follow are presented in 

10 terms of algorithms and symbolic representations of operations on data bits 

within a computer memory. These algorithmic descriptions and representations 
are the means used by those skilled in the data processing arts to most 
effectively convey the substance of their work to others skilled in the art. An 
algorithm is here, and generally, conceived to be a self-consistent sequence of 

15 steps leading to a desired result. The steps are those requiring physical 

manipulations of physical quantities. Usually, though not necessarily, these 
quantities take the form of electrical or magnetic signals capable of being stored, 
transferred, combined, compared, and otherwise manipulated. It has proven 
convenient at times, principally for reasons of common usage, to refer to these 

20 signals as bits, values, elements, symbols, characters, terms, numbers, or the like. 

It should be borne in mind, however, that all of these and similar terms 
are to be associated with the appropriate physical quantities and are merely 
convenient labels applied to these quantities. Unless specifically stated 
otherwise as apparent from the following discussion, it is appreciated that 

25 throughout the description, discussions utilizing terms such as "processing" or 
"computing" or "calculating" or "determining" or "displaying" or the like, refer to 
the action and processes of a computer system, or similar electronic computing 
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device, that manipulates and transforms data represented as physical 
(electronic) quantities within the computer system's registers and memories into 
other data similarly represented as physical quantities within the computer 
system memories or registers or other such information storage, transmission or 
5 display devices. 

The present invention also relates to apparatus for performing the 
operations herein. This apparatus may be specially constructed for the required 
purposes, or it may comprise a general-purpose computer selectively activated 
or reconfigured by a computer program stored in the computer. Such a 

10 computer program may be stored in a computer readable storage medium, such 
as, but is not limited to, any type of disk including floppy disks, optical disks, 
CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random 
access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or 
any type of media suitable for storing electronic instructions, and each coupled 

15 to a computer system bus. 

The algorithms and displays presented herein are not inherently related 
to any particular computer or other apparatus. Various general-purpose 
systems may be used with programs in accordance with the teachings herein, or 
it may prove convenient to construct more specialized apparatus to perform the 

20 required method steps. The required structure for a variety of these systems 

will appear from the description below. In addition, the present invention is not 
described with reference to any particular programming language. It will be 
appreciated that a variety of programming languages may be used to implement 
the teachings of the invention as described herein. 

25 
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Overview 

There is a need for a capability to allow the user interfaces of two or more 
applications to be combined or merged with one another. This capability allows 
5 for the free exchange of information between related applications and allows the 
context of a user interface from one application to be used by many different 
applications. 

A method and apparatus for melding user interfaces are described. 
Melded user interfaces combines the user interfaces of two or more applications 

10 and does not require the cooperation or acquiescence from the applications (or 
their programmers). Using melded user interfaces, the screen layout (e.g., base 
layout) corresponding to the user interface of one application may be used by 
one or more other applications to display data associated with that application. 
The application having the base layout is referred to herein as a parent 

15 application. The application whose data modifies the base layout of the parent 
application is referred to herein as a child application. There may be one or 
more children applications. The parent application provides the context for the 
displaying of the data from the children applications. 

In one embodiment, an overlay generation mechanism (e.g., application) 

20 retrieves raster data associated with the user interface of the parent application 
from the display buffer of a personal computer video card. The raster data is 
used to determine a layout so that the child application can overlay the data it 
wants to display. In one embodiment, in order to make the correct overlay, 
previously compiled information about the layout of the user interface of the 

25 parent application is utilized. For example, when the parent application is an 
appointment application displaying appointment information in a monthly 
calendar view, the information that can be compiled from the display includes 
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data such as, for example, the location of the month name, the locations of the 
day names, the locations of the day numbers, etc. It will be appreciated to note 
that the parent application's user interface is designed and provided by the 
software vendor, independent of any child application. This independence of 
5 the parent application from the child application is advantageous because it 
allows a wide range of children applications to be used with the parent 
application. 

Figure 1A is exemplary embodiment of a personal computer system 
that may be used to perform functionality described herein. The various 
10 components shown in Figure 1A are provided by way of example. Certain 
components of the computer in Figure 1A can be deleted from the addressing 
system for a particular implementation of the invention. The computer shown 
in Figure 1A may be any type of computer including a general-purpose 
computer. 

15 Figure 1A illustrates a system bus 10 to which various components are 

coupled. A processor 15 performs the processing tasks required by the 
computer. Processor 15 may be any type of processing device capable of 
implementing the steps necessary to perform the operations discussed above. 
An input/ output (I/O) device 20 is coupled to bus 10 and provides a 

20 mechanism for communicating with other devices coupled to the computer. A 
graphics display adapter 25 is connected to the bus to receive display data 
generated by the processor 15 and store the display data in a display buffer. A 
read-only memory (ROM) 30 and a random access memory (RAM) 35 are 
coupled to bus 10 and provide a storage mechanism for various data and 

25 information used by the computer, such as, for example, the overlay generation 
code and the pattern recognition code. Although ROM 30 and RAM 35 are 
shown coupled to bus 10, in alternate embodiments, ROM 30 and RAM 35 are 
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coupled directly to processor 15 or coupled to a dedicated memory bus (not 
shown). 

A video display 40 is coupled to the graphics display adapter 25 and 
displays various information and data stored in the display buffer to the user of 
5 the computer. The data display may include the base layout of the parent 
application by itself or with the overlay display from the child application. A 
disk drive 45 is coupled to bus 10 and provides for the long-term mass storage 
of information. Disk drive 45 may be used to store the parent application, the 
child application, and the overlay generation application. It may also be used to 

10 store data associated with the parent and the child application. A keyboard 50 
and a mouse 55 are provided to receive input from the user. 

Initially, the parent application is the active application that controls the 
information displayed on the video display 40 before the child application is 
activated. In one embodiment, the child application runs in the background and 

15 is activated by pressing a key or a key combination on the PC keyboard 50. It 
will be apparent to one skilled in the art that other methods can be used to 
activate the child application, such as, for example, positioning the pointer of the 
mouse 55 on an icon representing the child application and pressing the left 
mouse button. When the child application is activated, the display buffer is 

20 read, the pattern recognition operation is applied, and the overlay layout is 

generated. The user can then interact with the data from the child application in 
the context of the user interface of the parent application. 

In one embodiment, the child application continuously applies pattern 
recognition operations to the contents of the display buffer. When it detects the 

25 presence of a display format indicative of a known parent application, an 
indication of this event such as, for example, a beep or a flashing icon, is 
displayed to the user. This indication can be parameterized by the amount of 
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data the child can display in the current context of the parent. The user can then 
choose to invoke the child application in the manner described above. In 
another embodiment, the child application may be invoked automatically after a 
defined period of inactivity by the user. 
5 Figure IB is a flow diagram illustrating an exemplary embodiment of the 

present invention. At block 105, the raster data is read from the PC graphics 
card. The raster data is analyzed by a pattern recognition operation, as shown 
in block 110. The pattern recognition operation looks for patterns in the raster 
data to identify all or portions of a layout. In one embodiment, information 

10 previously compiled about the user interface of the parent application is utilized 
to identify and locate different sections of the layout, as shown in block 115. For 
example, when using a calendar display as the display from the parent 
application, the information about the calendar may comprise of the format of 
the calendar, the locations of the day names and the locations of the day 

15 numbers on the calendar. As another example, the display from the parent 
application is the window file system layout, and the information about the 
window file system may comprise the format of the file tree, the locations of the 
icons representing the directories, etc. 

The layout and the previously compiled information about the 

20 corresponding user interface are used by the overlay generation mechanism 
(e.g., application) to generate an overlay layout, as shown in block 120. In one 
embodiment, the overlay may comprise information from both the parent 
application and the child application. At block 125, the overlay is written into 
the display buffer and presented to the user through a display monitor. 

25 At block 130, the user is able to interact with the child application 

through the melded user interface generated using the overlay. Using the 
layout of the parent application to display the data from the child application 
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provides the impression that both the parent application and the child 
application are integrated with one another. In one embodiment, the user of the 
parent application does not have to learn a new user interface to interact with 
the child application. 

5 In one embodiment, when the child application is invoked, it writes data 

into the display buffer. The child application knows where (e.g., x, y pixel 
location) in the display buffer each data item was written. It also sets an event 
mask in the operating system that intercepts events from the user interface 
devices (e.g., keyboard, mouse, etc.). Such events typically include an 

10 identification of the event (e.g., identity of the key pressed, mouse button 
clicked, or x-y cursor position). 

Based on a combination of event identity, its x-y location, and the 
locations of the child's data items, the event is either processed by the child 
application or it is processed by the operating system as it would normally be if 

15 the child application were not present. This allows the user to select the data 
displayed by the child application. For example, the user can move the mouse 
cursor on top of data items that were written in the display buffer by the child 
and click a mouse button to display another data item. 

Figure 2 is an exemplary calendar display that can be used with the 

20 present invention. The calendar display is from a web calendar manager which 
runs as the parent application. Information about the layout of the calendar 
display includes the knowledge that the calendar manager runs from within a 
web browser, and the calendar manager typically displays monthly calendar 
views, characterized by a rectangular grid layout with one grid cell used for 

25 each day of a month. Furthermore the month and year are displayed along the 
top center area 205, the names of the days are displayed in the first row 210, 
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centered above a calendar grid cell, and the day number of each day is 
displayed in the upper right corner 215 of each grid cell 

Using the knowledge about the parent application, the overlay 
generation application can generate its overlay on the layout of the parent 
5 application without altering the user interface of the parent application or 
causing the user to learn a new user interface. 

In the calendar display example, the display from the calendar manager 
can be integrated with information from a child application, such as, for 
example, a document manager application. The document manager application 

10 retrieves documents previously generated and saved in a storage device. The 
overlay generation mechanism (e.g., application) uses the precompiled 
knowledge about the calendar manager (e.g., parent application) and generates 
an overlay for the documents retrieved by the document manager (e.g., child 
application) in the context of the parent application. For example, all documents 

15 to be reviewed by the user on September 15, 1999 are shown on the calendar 
display within the grid identifying the date September 15, 1999. 

The pattern recognition operation detects whether the display buffer 
contains data representing a monthly calendar view. It finds out where the 
calendar is on the display and calculates the coordinates of the grid cell for each 

20 day. From the coordinates of the grid cells, a grid layout of the entire calendar 
can be generated. The pattern recognition operation is performed with a series 
of standard computer vision or image processing operation, which includes an 
edge detection operation. Figure 3A is an exemplary layout result produced by 
the edge detection operation. Note that the highlighted edges correspond to the 

25 edges shown in Figure 2. It will be apparent to one skilled in the art that other 
pattern recognition operations can be used to generate the grid layout without 
deviating from the invention. 
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The edges of the layout result shown in Figure 3A are filtered through a 
smoothing operation. This smoothing operation groups the pixels and connects 
the lines together. Figure 3B is an exemplary result produced by the smoothing 
operation. A line detection algorithm is then applied to locate the boundaries of 
5 the grid cells, as shown in Figure 4A. The line intersections and corners are 
identified using standard techniques, as shown in Figure 4B. This provides the 
grid for the calendar display. 

The precompiled information about the calendar display is then used to 
estimate the locations of the day number, the day names, the months, and the 

10 year information in the grid of the calendar, as shown in Figure 4C In one 

embodiment, optical character recognition (OCR) is applied to the raster data in 
these locations to obtain the day number, the month and the year displayed in 
the calendar. Contextual post-processing using knowledge about the calendar is 
applied to verify the character recognition results. 

15 The overlay generation application uses the grid information and the 

knowledge about the locations of the day number, the day and the month to 
generate the overlay corresponding to the lay out of the parent application. 
Figure 5A illustrates an exemplary calendar parent application with the 
overlay information from the child application. In one embodiment, the 

20 document manager (i.e., child application) uses icons or thumbnail images to 
represent the documents. The physical location on the screen where the 
thumbnail images are placed is determined based on the grid cell locations and 
dates that were found by the pattern recognition operation. The document 
thumbnail images 505 are then included in the overlay and displayed in the 

25 melded user interface. 

In one embodiment, the user may dynamically adjust the size of the 
icons. In another embodiment, the overlay generation operation extracts the 
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first few pages of the documents and displays them in the melded user interface. 
Alternatively, the title or some other information indicative of the documents 
may be displayed instead of the thumbnail images or icons. 

The overlay generation mechanism (e.g., application) may have to 
5 determine the locations of the data displayed by the parent application so that 
the placements of the thumbnail images from the child application do not 
overlap with the data from the parent application. In one embodiment, the 
overlay generation application may display the document thumbnail images on 
top of the data displayed by the parent application. The user may invoke the 
10 overlay mode by pressing a function key on the PC keyboard. This triggers the 
execution of the child application and displays the thumbnail images on the 
calendar. 

Figure 5B illustrates an exemplary interaction with the child application 
though the melded user interfaces. Through the melded user interface, the user 

15 interacts with the child application by selecting the thumbnail images. Clicking 
on the thumbnail images 510 navigates the user to higher resolution documents 
represented by the thumbnail images. In one embodiment, the icons are also 
hotlinked to complete document descriptions 515 so that when the user places 
the cursor over the icon the complete description is displayed. 

20 In one embodiment, the thumbnail images displayed in a grid cell show 

one or more documents that were recorded on that date. The display of the 
parent application and the child application can be toggled from one to the 
other. When the display from the child application is toggled on, the 
information from the parent application may be suppressed or overlapped by 

25 the information from the child application. 
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Also, when the display from the child application is toggled on, the child 
application has control of the interaction between the user and the system. On 
the other hand, when the display from the parent application is toggled on, the 
information from the parent application reappears and the parent application 
5 has control of the interaction between the user and the system. 

When the document thumbnail image is selected, the document is 
retrieved by the child application. The child intercepts the "selection," as 
previously described and retrieves the document based on the event type. For 
example, clicking on the left mouse button could retrieve (from a document 
10 server), a high resolution image of the document depicted in the thumbnail The 
child maintains a table of network addresses (e.g., URL's) that correspond to 
each thumbnail and event type. Those network addresses are accessed and their 
contents retrieved when the particular event is executed. 

Alternatively, the document can be retrieved by another application 
15 outside of the child application. The document may be retrieved from a 

document server through a network connection. In one embodiment, when 
there are multiple documents to be displayed in the same grid cell, only the first 
few pages of the document are shown as representative pages. An indicator 
such as, for example, a green bar or a number is used to indicate to the user that 
20 the document has additional pages to be seen. This enables the user to go and 
look further. 

In another embodiment, the text displayed by the parent application can 
be used to initiate a document retrieval request to the document manager. For 
example, when an appointment in the calendar display is a birth day 
25 appointment, the child application may retrieve all documents related to people 
having birth days on that particular date. 
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In another embodiment, the user interface can depend on the information 
provided by operating system. For example, with the web calendar as the 
parent application running in a browser, the child application may analyze the 
uniform resource locations (URL) associated with the web calendar application. 
5 The URL may provide information that lets the child application know about 
the type of calendar view being displayed (e.g., week grid, month grid) without 
having to use the OCR. In another embodiment, both of the information from 
the registry and the OCR results may be used. 

In the foregoing discussion, the document manager application is used to 

10 illustrate a child application overlaying the display of the calendar display 
parent application. Other children applications can also be used to take 
advantage of the present invention. For example, trip information, airline 
reservation information, and hotel confirmation information can be generated in 
the overlay by the overlay generation application using the display buffer. It 

15 will be apparent to one skilled in the art that other applications can also be used 
as the parent application. 

Figure 6 illustrates an embodiment of a computer-readable medium 600 
containing various sets of instructions, code sequences, configuration 
information, and other data used by a computer or other processing device. The 

20 embodiment illustrated in Figure 6 is suitable for use with the melded user 

interface method described above. The various information stored on medium 
600 is used to perform various data processing operations. Computer-readable 
medium 600 is also referred to as a processor-readable medium. Computer- 
readable medium 600 can be any type of magnetic, optical, or electrical storage 

25 medium including a diskette, magnetic tape, CD-ROM, memory device, or other 
storage medium. 
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Computer-readable medium 600 includes interface code 602 that controls 
the flow of information between various devices or components in the computer 
system. Interface code 602 may control the transfer of information within a 
device (e.g., between the processor and a memory device), or between an 
5 input/ output port and a storage device. Additionally, interface code 602 may 
control the transfer of information from one device to another. 

Computer-readable medium 600 also includes the overlay generation 
application 604 that is used to generate the overlay. Other codes stored on the 
computer-readable medium 600 may include the pattern recognition code 606, 
10 the edge smoothing code 608, and the optical character recognition code 612. 

From the above description and drawings, it will be understood by those 
of ordinary skill in the art that the particular embodiments shown and described 
are for purposes of illustration only and are not intended to limit the scope of 
the invention. Those of ordinary skill in the art will recognize that the invention 
15 may be embodied in other specific forms without departing from its spirit or 

essential characteristics. References to details of particular embodiments are not 
intended to limit the scope of the claims. 
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CLAIMS 

What is claimed is: 



1 L A method comprising: 

2 extracting a first data from a display buffer, the first data generated by a 

3 first application and being associated with a user interface from the 

4 first application; 

5 recognizing a layout from the first data; and 

6 using the layout to create an overlay to display a second data generated by 

7 a second application, wherein there is no direct link between the first 

8 application and the second application. 

1 2. The method of claim 1, wherein recognizing the layout comprises 

2 performing a pattern recognition operation on the first data to create the 

3 layout. 

1 3. The method of claim 1, wherein using the layout to create the overlay 

2 comprises: 

3 determining an overlay location on the layout to place the second data 

4 based on known information about the layout; 

5 generating the overlay of the layout; 

6 placing the second data in the overlay; and 

7 merging the overlay with the layout. 
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1 4. The method of claim 3, wherein the overlay location has a context 

2 consistent with the second data. 

1 5. The method of claim 4, wherein the context is provided by the first 

2 application, and wherein a user interacts with the second application 

3 using the context. 

1 6. The method of claim 1, further comprising: 

2 writing the overlay in the display buffer such that the second data is 

3 displayed at the overlay location without changing sections of the first 

4 data outside of the overlay location; 

5 displaying information in the display buffer; and 

6 interacting with the second application through the second data at the 

7 overlay location. 

1 7. The method of claim 6, further comprising running the first application in 

2 the background while interacting with the second application. 

1 8. The method of claim 1, wherein the first application runs independently 

2 from the second application. 
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1 9. A machine-readable medium providing instructions, which when executed 

2 by a set of one or more processors, cause said set of processors to perform 

3 the following: 

4 extracting a first data from a display buffer, the first data generated by a 

5 first application and being associated with a user interface from the 

6 first application; 

7 recognizing a layout from the first data; and 

8 using the layout to create an overlay to display a second data generated by 

9 a second application, wherein there is no direct link between the first 
10 application and the second application. 

1 10. The machine-readable medium of claim 9, wherein recognizing the layout 

2 comprises performing a pattern recognition operation on the first data to 

3 create the layout. 

1 11. The machine-readable medium of claim 9, wherein using the layout to 

2 create the overlay comprises: 

3 determining an overlay location on the layout to place the second data 

4 based on known information about the layout; 

5 generating the overlay of the layout; 

6 placing the second data in the overlay; and 

7 merging the overlay with the layout. 
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1 12. The machine-readable medium of claim 11, wherein the overlay location 

2 has a context consistent with the second data. 

1 13. The machine-readable medium of claim 12, wherein the context is provided 

2 by the first application, and wherein a user interacts with the second 

3 application using the context. 

1 14. The machine-readable medium of claim 9, further comprising: 

2 writing the overlay in the display buffer such that the second data is 

3 displayed at the overlay location without changing sections of the first 

4 data outside of the overlay location; 

5 displaying information in the display buffer; and 

6 interacting with the second application through the second data at the 

7 overlay location. 

1 15. The machine-readable medium of claim 14, further comprising running the 

2 first application in the background while interacting with the second 

3 application. 

1 16. The machine-readable medium of claim 9, wherein the first application 

2 runs independently from the second application. 

1 17. A computer system, comprising: 

2 a bus; 



Ricoh 



23 



074451. PI 15 



3 a data storage device coupled to the bus; and 

4 a processor coupled to the data storage device, the processor operable 

5 to receive instructions which, when executed by the processor, cause 

6 the processor to perform a method comprising: 

7 extracting a first data from a display buffer, the first data generated by 

8 a first application and being associated with a user interface from 

9 the first application; 

10 recognizing a layout from the first data; and 

1 1 using the layout to create an overlay to display a second data 

12 generated by a second application, wherein there is no direct link 

13 between the first application and the second application. 

1 18. The system of claim 17, wherein recognizing the layout comprises 

2 performing a pattern recognition operation on the first data to create the 

3 layout. 

1 19. The system of claim 17, wherein using the layout to create the overlay 

2 comprises: 

3 determining an overlay location on the layout to place the second data 

4 based on known information about the layout; 

5 generating the overlay of the layout; 

6 placing the second data in the overlay; and 

7 merging the overlay with the layout. 
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1 20. The system of claim 19, wherein the overlay location has a context 

2 consistent with the second data, 

1 21. The system of claim 20, wherein the context is provided by the first 

2 application, and wherein a user interacts with the second application 

3 using the context. 

1 22. The system of claim 17, further comprising: 

2 writing the overlay in the display buffer such that the second data is 

3 displayed at the overlay location without changing sections of the first 

4 data outside of the overlay location; 

5 displaying information in the display buffer; and 

6 interacting with the second application through the second data at the 

7 overlay location. 

1 23. The system of claim 22, further comprising running the first application in 

2 the background while interacting with the second application. 

1 24. The system of claim 17, wherein the first application runs independently 

2 from the second application. 
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1 25. A method, comprising: 



2 modifying data in a display buffer that is generated by a first application 

3 with data generated by a second application, the first application 

4 running independently from the second application; and 

5 receiving input in response to user interactions with the second application 

6 through a user interface associated with the data generated by the first 

7 application, wherein the data generated by the second application is 

8 placed in a location in the user interface, wherein the location is 

9 contextually consistent with the data generated by the second 
10 application. 

1 26. The method of claim 25, wherein modifying data in the display buffer 

2 comprises: 

3 performing a pattern recognition operation on the data generated by the 

4 first application to create a layout; and 

5 forming an overlay with the layout and with predetermined information 

6 about a display corresponding to the user interface, the overlay used 

7 to determine placement of the data generated by the second 

8 application in the display. 

1 27. The method of claim 26, wherein the layout comprises of grid cells 

2 corresponding to display areas in the user interface, and wherein the data 

3 generated by the second application is placed in the grid cells. 
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1 28. The method of claim 25, wherein the first application runs in the 

2 background while the user interacts with the second application. 



1 29. A machine-readable medium providing instructions, which when executed 

2 by a set of one or more processors, cause said set of processors to perform 

3 the following: 

4 modifying data in a display buffer that is generated by a first application 

5 with data generated by a second application, the first application 

6 running independently from the second application; and 

7 receiving input in response to user interactions with the second application 

8 through a user interface associated with the data generated by the first 

9 application, wherein the data generated by the second application is 

10 placed in a location in the user interface, wherein the location is 

1 1 contextually consistent with the data generated by the second 

12 application. 

1 30. The machine-readable medium of claim 29, wherein modifying data in the 

2 display buffer comprises: 

3 performing a pattern recognition operation on the data generated by the 

4 first application to create a layout; and 

5 forming an overlay with the layout and with predetermined information 

6 about a display corresponding to the user interface, the overlay used 

7 to determine placement of the data generated by the second 

8 application in the display. 
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1 31. The machine-readable medium of claim 30, wherein the layout comprises 

2 of grid cells corresponding to display areas in the user interface, and 

3 wherein the data generated by the second application is placed in the grid 

4 cells. 

1 32. The machine-readable medium of claim 29, wherein the first application 

2 runs in the background while the user interacts with the second 

3 application. 

1 33. A computer system, comprising: 

2 a bus; 

3 a data storage device coupled to the bus ; and 

4 a processor coupled to the data storage device, the processor operable 

5 to receive instructions which, when executed by the processor, cause 

6 the processor to perform a method comprising: 

7 modifying data in a display buffer that is generated by a first 

8 application with data generated by a second application, the first 

9 application running independently from the second application; 

10 and 

1 1 receiving input in response to user interactions with the second 

12 application through a user interface associated with the data 

13 generated by the first application, wherein the data generated by 

14 the second application is placed in a location in the user 
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15 interface, wherein the location is contextually consistent with the 

1 6 data generated by the second application. 

1 34. The computer system of claim 33, wherein modifying data in the display 

2 buffer comprises: 

3 performing a pattern recognition operation on the data generated by the 

4 first application to create a layout; and 

5 forming an overlay with the layoutand with predetermined information 

6 about a display corresponding to the user interface, the overlay used 

7 to determine placement of the data generated by the second 

8 application in the display. 

1 35. The computer system of claim 34, wherein the layout comprises of grid 

2 cells corresponding to display areas in the user interface, and wherein the 

3 data generated by the second application is placed in the grid cells. 

1 36. The computer system of claim 33, wherein the first application runs in the 

2 background while the user interacts with the second application. 

1 37. A method comprising: 

2 reading raster data from a raster display buffer containing an image 

3 generated by a first application; 

4 performing a pattern recognition on the image to generate a pattern; 

5 applying predetermined information about the image with the pattern to 

6 determine a layout of the image; 
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7 



8 



generating an overlay using the layout of the image; and 
placing data generated by a second application on the overlay. 



1 38. The method of claim 37, further comprising writing the overlay into the 

2 raster display buffer. 

1 39. The method of claim 37, wherein the image comprises a user interface 

2 from the first application, and wherein a user interacts with the second 

3 application through the user interface while the first application runs in 

4 the background. 

The method of claim 39, wherein while the user interacts with the second 
application, the first application has no control of input received from the 
user. 



1 40. 

2 
3 
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ABSTRACT OF THE DISCLOSURE 



In one embodiment, a method for combining the user interfaces of several 
applications is disclosed. Data generated by a first application is extracted from 
a display buffer. The data is associated with a user interface from the first 
application. From the extracted data, a layout pattern is recognized. Using the 
layout, an overlay is created. The overlay is used to display a second data 
generated by a second application. There is no direct link between the first 
application and the second application. 
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DECLARATION AND POWER OF ATTORNEY FOR PATENT APPLICATION 



As a beiow named inventor, I hereby declare that: 

My residence, post office address and citizenship are as stated below, next to my name. 

I believe I am the original, first, and sole inventor (if only one name is listed below) or an original, 
first, and joint inventor (if plural names are listed below) of the subject matter which is claimed and 
for which a patent is sought on the invention entitled 

MELDED USER INTERFACES 



the specification of which 

x is attached hereto. 

was filed on . _as 

United States Application Number 

or PCT International Application Number 

and was amended on ■ 

(if applicable) 

I hereby state that I have reviewed and understand the contents of the above-identified 
specification, including the claim(s), as amended by any amendment referred to above. I do not 
know and do not believe that the claimed invention was ever known or used in the United States of 
America before my invention thereof, or patented or described in any printed publication in any 
country before my invention thereof or more than one year prior to this application, that the same 
was not in public use or on sale in the United States of America more than one year prior to this 
application, and that the invention has not been patented or made the subject of an inventor's 
certificate issued before the date of this application in any country foreign to the United States of 
America on an application filed by me or my legal representatives or assigns more than twelve 
months (for a utility patent application) or six months (for a design patent application) prior to this 
application. 

I acknowledge the duty to disclose all information known to me to be material to patentability as 
defined in Title 37, Code of Federal Regulations, Section 1 .56. 

I hereby claim foreign priority benefits under Title 35, United States Code, Section 1 19(a)-(d), of any 
foreign application(s) for patent or inventor's certificate listed below and have also identified beiow 
any foreign application for patent or inventor's certificate having a filing date before that of the 
application on which priority is claimed: 
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Prior Foreign Application^ 



Priority 
Claimed 



(Number) (Country) (Day/Month/Year Filed) Yes No 



(Number) (Country) (Day/Month/Year Filed) Yes No 



(Number) (Country) (Day/Month/Year Filed) Yes No 

I hereby claim the benefit under title 35, United States Code, Section 1 1 9(e) of any United States 
provisional application(s) listed below: 



(Application Number) Filing Date 



(Application Number) Filing Date 



I hereby claim the benefit under Title 35, United States Code, Section 120 of any United States 
application(s) listed below and, insofar as the subject matter of each of the claims of this application 
is not disclosed in the prior United States application in the manner provided by the first paragraph 
of Title 35, United States Code, Section 1 12, 1 acknowledge the duty to disclose all information 
known to me to be material to patentability as defined in Title 37, Code of Federal Regulations, 
Section 1 .56 which became available between the filing date of the prior application and the national 
or PCT international filing date of this application: 



(Application Number) Filing Date (Status -- patented, 

pending, abandoned) 



(Application Number) Filing Date (Status -- patented, 

pending, abandoned) 

I hereby appoint the persons listed on Appendix A hereto (which is incorporated by reference and a 
part of this document) as my respective patent attorneys and patent agents, with full power of 
substitution and revocation, to prosecute this application and to transact all business in the Patent 
and Trademark Office connected herewith. 

Send correspondence to Michael J. Mallie , BLAKELY, SOKOLOFF, TAYLOR & 

(Name of Attorney or Agent) 
ZAFMAN LLP, 12400 Wilshire Boulevard 7th Floor, Los Angeles, California 90025 and direct 

telephone calls to Michael J. Mallie , (408) 720-8300. 

(Name of Attorney or Agent) 
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I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United 
States Code and that such willful false statements may jeopardize the validity of the 
application or any patent issued thereon. 



Full Name of Sole/First Inventor Jonathan J. Hull 
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9 ' Date _jM*_ 



Residence San Carlos. California Citizenship LLSA 

(City, State) (jUt) (Country) 

Post Office Address 754 N fLaurel Street fit . 

San Carlos, CA 94070 . 



Full Name of Second/Joint Inventor Peter Hart 




Inventor's Signature firyV^^&'^^ C^ Date 



Residence Menlo Park. California Citizenship LLSA — 

(City, State) (Country) 

Post Office Address 301 Arbor Road 



Menlo Park. CA 94025 



Full Name of Third/Joint Inventor . 



Inventor's Signature Date . 

Residence . Citizenship . 



(City, State) (Country) 
Post Office Address 
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Inventor's Signature . Date 

Residence . Citizenship 

(City, State) (Country) 

Post Office Address 
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APPENDIX A 



William E. Alford, Reg. No. 37,764; Farzad E. Amini, Reg. No. P42,261 ; Aloysius T. C. AuYeung, Reg. No. 
35 f 432; William Thomas Babbitt, Reg. No. 39,591; Carol F. Barry, Reg. No. 41,600; Jordan Michael 
Becker, Reg. No. 39,602; Bradley J. Bereznak, Reg. No. 33,474; Michael A. Bernadicou, Reg. No. 35,934; 
Roger W. Blakely, Jr., Reg. No. 25,831; Gregory D. Caldwell, Reg. No. 39,926; Ronald C. Card, Reg. No. 
44,587; Andrew C. Chen, Reg. No. 43,544; Thomas M. Coester, Reg. No. 39,637; Alin Corie, Reg. No. 
P46,244; Dennis M. deGuzman, Reg. No. 41,702; Stephen M. De Klerk, under 37 C.F.R. § 10.9(b); 
Michael Anthony DeSanctis, Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; Robert Andrew Diehi, 
Reg. No. 40,992; Sanjeet Dutta, Reg. No. P46,145; Matthew C. Fagan, Reg. No. 37,542; Tarek N. Fahmi, 
Reg. No. 41,402; Paramita Ghosh, Reg. No. 42,806; James Y. Go, Reg. No. 40,621; James A. Henry, 
Reg. No. 41,064; Wilimore F. Hoibrow III, Reg. No. P41,845; Sheryl Sue Holloway, Reg. No. 37,850; 
George W Hoover II, Reg. No. 32,992; Eric S. Hyman, Reg. No. 30,139; William W. Kidd, Reg. No. 
31,772; Sang Hui Kim, Reg. No. 40,450; Eric T. King, Reg. No. 44,188; Erica W. Kuo, Reg. No. 42,775; 
Kurt P. Leyendecker, Reg. No. 42,799; Michael J. Mallie, Reg. No. 36,591; Andre L. Marais, under 37 
C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. 42,879; Darren J. Milliken, Reg. 42,004; Lisa A. Norris, 
Reg. No. 44,976; Chun M. Ng, Reg. No. 36,878; Thien T. Nguyen, Reg. No. 43,835; Thinh V. Nguyen, 
Reg. No. 42,034; Dennis A. Nicholls, Reg. No. 42,036; Daniel E. Ovanezian, Reg. No. 41,236; Marina 
Portnova, Reg. No. P45,750; Babak Redjaian, Reg. No. 42,096; William F. Ryann, Reg. 44,313; James 
H. Salter, Reg. No. 35,668; William W. Schaal, Reg. No. 39,018; James C. Scheller, Reg. No. 31 ,1 95; 
Jeffrey Sam Smith, Reg. No. 39,377; Maria McCormack Sobrino, Reg. No. 31,639; Stanley W. Sokoloff, 
Reg. No. 25,128; Judith A. Szepesi, Reg. No. 39,393; Vincent P. Tassinari, Reg. No. 42,179; Edwin H. 
Taylor, Reg. No. 25,129; John F. Travis, Reg. No. 43,203; George G. C. Tseng, Reg. No. 41,355; Joseph 
A. Twarowski, Reg. No. 42,191; Lester J. Vincent, Reg. No. 31,460; Glenn E. Von Tersch, Reg. No. 
41,364; John Patrick Ward, Reg. No. 40,216; Mark L. Watson, Reg. No. P46,322; Thomas C. Webster, 
Reg. No. P46,154; Charles T. J. Weigell, Reg. No. 43,398; Kirk D. Williams, Reg. No. 42,229; James M. 
Wu, Reg. No. 45,241; Steven D. Yates, Reg. No. 42,242; and Norman Zafman, Reg. No. 26,250; my 
patent attorneys, and Justin M. Dillon, Reg. No. 42,486; my patent agent, of BLAKELY, SOKOLOFF, 
TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard, 7th Floor, Los Angeles, 
California 90025, telephone (310) 207-3800, and James R. Thein, Reg. No. 31,710, my patent attorney. 
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APPENDIX B 



Title 37, Code of Federal Regulations, Section 1 .56 
Duty to Disclose Information Material to Patentability 

(a) A patent by its very nature is affected with a public interest. The public interest is best served, 
and the most effective patent examination occurs when, at the time an application is being examined, the 
Office is aware of and evaluates the teachings of ail information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that individual 
to be material to patentability as defined in this section. The duty to disclosure information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit information which is not material 
to the patentability of any existing claim. The duty to disclosure all information known to be material to 
patentability is deemed to be satisfied if all information known to be material to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1.98. However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1 ) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1 ) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 
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